Skip to content

fix(orchestration): scope coordinator runs/tasks/dispatches by workspace (#4389) - #6587

Draft
nwparker wants to merge 7 commits into
mainfrom
nwparker/fix-4389
Draft

nwparker wants to merge 7 commits into
mainfrom
nwparker/fix-4389

Conversation

@nwparker

@nwparker nwparker commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #4389. Orca stored orchestration state in one runtime-wide database but did not record which workspace owned each coordinator, task, dispatch, or message. As a result, a coordinator in workspace B could be rejected because A was running, a B-scoped stop could stop A, and concurrent loops could count, dispatch, or attribute each other's work.

This PR:

  • adds nullable workspace_key ownership to coordinator runs, tasks, dispatch contexts, and messages through the existing transactional SQLite migration (v5 → v6);
  • resolves selectors through Orca's provider-agnostic worktree resolver, so local and SSH/remote-backed worktrees use the same canonical scope;
  • scopes coordinator polling, convergence, concurrency counts, stale-dispatch checks, messages, and decision-gate repair to the owning workspace;
  • replaces the module-global coordinator singleton with a workspace-keyed map;
  • makes run-stop --worktree <selector> exact, keeps --worktree all as explicit global mode, and rejects an ambiguous bare run-stop;
  • preserves current-main stale terminal-handle validation after rebasing; and
  • treats every concurrently active coordinator prompt as user-owned, preventing push-on-idle from auto-submitting into an earlier workspace's coordinator just because another run is newer.

No UI, keyboard, git-provider, shell, or local-path-specific behavior is introduced.

Evidence of fix

Before, on current origin/main (c5669cceb5), a production RpcDispatcher reproduction with two in-memory coordinator fixtures failed both user-visible invariants:

$ pnpm exec vitest run --config config/vitest.config.ts \
    src/main/runtime/rpc/methods/orchestration-4389-repro.test.ts

Test Files  1 failed (1)
Tests       2 failed (2)

expected startB.ok true, received false
expected B stop run ID != A run ID, received the same run ID

The exact reproduction and expected/actual behavior are also recorded on the issue: #4389 (comment).

After, the permanent production-RPC regression proves B can start while A runs and a B stop leaves A active:

$ pnpm exec vitest run --config config/vitest.config.ts \
    src/main/runtime/rpc/methods/orchestration-gates.test.ts \
    -t 'does not reject a run in worktree B|runStop for worktree B leaves worktree A running'

Test Files  1 passed (1)
Tests       2 passed | 7 skipped (9)

Additional verification on the rebased branch:

$ pnpm exec vitest run --config config/vitest.config.ts \
    src/main/runtime/rpc/methods/orchestration-gates.test.ts \
    src/main/runtime/orchestration/db.test.ts \
    src/main/runtime/runtime-rpc.test.ts \
    src/main/runtime/orca-runtime.test.ts \
    src/main/runtime/orchestration/coordinator.test.ts \
    src/main/runtime/orchestration/formatter.test.ts \
    src/cli/index.test.ts src/cli/handlers/orchestration.test.ts

Test Files  8 passed (8)
Tests       915 passed (915)

$ pnpm run typecheck:cli && pnpm run typecheck:node
# passed

$ git diff --name-only origin/main -- '*.ts' | xargs pnpm exec oxlint
# passed

$ pnpm check:max-lines-ratchet
max-lines ratchet OK — 355 grandfathered suppression(s), no new bypasses.

$ pnpm build
# desktop, relay, CLI, Electron/Vite, web, and native builds passed

Reliability proof:

  • Class/invariant: agent-session.coordinator-ownership; a task, dispatch, exit escalation, status row, injected message, and stop request must resolve only to the coordinator owning that workspace.
  • Oracle: deterministic RPC tests start A and B together, stop B, and assert A remains active; a runtime regression test also proves an earlier active coordinator never receives an automatic Enter when a newer workspace run exists.
  • Product change type: runtime hardening plus deterministic migration/ownership tests.
  • Manifest status: config/reliability-gates.jsonc does not exist on current main; the intended partial gate is orchestration.workspace-coordinator-isolation, represented by the blocking unit/RPC suites above.
  • Provider/platform coverage: selector resolution is shared by local and SSH/remote worktrees; no PTY provider operation, shell command, path separator, or platform branch changed. Local macOS and cross-platform build/type contracts were exercised; live SSH/WSL/mobile relay and live Electron multi-worktree runs remain unproved because the failure is fully observable below those layers.
  • Performance budget: no polling loop, provider listing, subprocess, startup await, or renderer work was added. Poll-time queries are narrowed by workspace and indexed; active-coordinator prompt lookup has a (coordinator_handle, status) index, including after migration.
  • Rollback/demotion rule: if scoped selectors widen to global state, concurrent RPC cases cross-return a run ID, or the migration/index test fails, the partial gate should be treated as failed and the feature reverted or corrected before release.

ELI5

Two managers were sharing one unlabeled pile of jobs. Manager B could be told “a manager is already working,” even though that was manager A in another workspace; asking B to stop could then stop A. Orca now labels each manager and job with its workspace, so each manager reads, assigns, reports, and stops only its own work.

User-facing trade-offs

  • Explicit invalid or ambiguous worktree selectors now fail closed instead of silently becoming a global coordinator.
  • A bare run-stop is convenient only when one run is active; with multiple runs, users must identify the workspace (or use --worktree all for the global run).
  • A global coordinator intentionally cannot run beside scoped coordinators because global mode can see every task.
  • Existing pre-v6 rows keep NULL ownership and remain visible as legacy/global data for upgrade compatibility; newly created tasks and runs are isolated.
  • The migration adds four nullable columns and a few small indexes, increasing the orchestration database slightly.
  • Starting/stopping a scoped run and creating a scoped task each perform one canonical workspace lookup; no extra work occurs on terminal output, rendering, or unrelated UI paths.

@coderabbitai

coderabbitai Bot commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds workspace-scoped concurrent orchestration coordinator support. A new workspace_key column is added to four orchestration SQLite tables (messages, tasks, dispatch_contexts, coordinator_runs) with a v5→v6 migration. All DB read/write methods gain optional workspaceKey scoping via a shared workspaceScopeClause helper. The Coordinator class propagates workspaceKey through its polling loops, decision-gate processing, and task ownership checks. The RPC layer replaces a single activeCoordinator with an activeCoordinators map keyed per workspace. orchestration.runStop gains an optional worktree parameter to select which coordinator to stop. The CLI exposes --worktree <selector|all> on both orchestration run and orchestration run-stop, with corresponding spec, help text, and documentation updates. Two new public runtime methods (resolveWorkspaceKeyForSelector, resolveWorkspaceKeyForTerminalHandle) resolve worktree selectors to workspace keys.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The PR description is detailed, but it does not follow the required template and is missing Summary, Screenshots, Testing, AI Review Report, Security Audit, and Notes sections. Reformat the description to match the template and add the missing sections, especially the testing checklist plus AI review and security audit summaries.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the workspace-scoping fix for coordinator runs, tasks, and dispatches.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nwparker

Copy link
Copy Markdown
Contributor Author

Evidence — #4389 regression proof

Root cause confirmed in the code

  • getActiveCoordinatorRun() (db.ts) returned the single most-recent status='running' run with no workspace filter.
  • orchestration-gates.ts used one module-level let activeCoordinator; a 2nd orchestration.run overwrote it (orphaning the first) and runStop stopped whoever owned it.
  • The coordinator loop's listTasks({ready}), listTasks({status:'dispatched'}), checkConvergence(), and getStaleDispatches() were all unscoped → cross-run interference.
  • failActiveDispatchOnExit, agent-status, and lineage all called the unscoped getActiveCoordinatorRun() → cross-attribution.

After the fix — targeted test runs (actual output)

Data-layer scoping + start/stop guard:

$ vitest run src/main/runtime/orchestration/db.test.ts \
             src/main/runtime/rpc/methods/orchestration-gates.test.ts --reporter=verbose

✓ db.test.ts > workspace scoping (#4389) > getActiveCoordinatorRun returns each workspace its own active run
✓ db.test.ts > workspace scoping (#4389) > a legacy (NULL) run is visible to every scoped lookup but a scoped run is not
✓ db.test.ts > workspace scoping (#4389) > listTasks scoped to A never returns B rows (legacy NULL stays shared)
✓ db.test.ts > workspace scoping (#4389) > getStaleDispatches scoped to A never surfaces B dispatches
✓ db.test.ts > workspace scoping (#4389) > getIdleTerminals scoped to A treats B-workspace busy terminals as available
✓ orchestration-gates.test.ts > start/stop guard scoping (#4389) > does not reject a run in worktree B while a run is active in worktree A
✓ orchestration-gates.test.ts > start/stop guard scoping (#4389) > rejects a second run in the SAME worktree
✓ orchestration-gates.test.ts > start/stop guard scoping (#4389) > runStop for worktree B leaves worktree A running

 Test Files  2 passed (2)
      Tests  60 passed (60)

v5 → v6 migration:

$ vitest run src/main/runtime/runtime-rpc.test.ts -t "migration"
✓ orchestration DB migration (§3.2) > upgrades a v5 on-disk DB to v6 with workspace_key columns, preserving rows as NULL
 Test Files  1 passed (1)
      Tests  3 passed | 36 skipped (39)

Full affected-suite sweep (no regressions):

$ vitest run src/main/runtime/orchestration/ src/main/runtime/runtime-rpc.test.ts \
    src/main/runtime/orca-runtime.test.ts src/main/runtime/rpc/methods/orchestration-gates.test.ts \
    src/cli/handlers/orchestration.test.ts
 Test Files  9 passed (9)
      Tests  677 passed (677)
$ vitest run src/main/runtime/rpc/      # all RPC method tests
 Test Files  46 passed (46)
      Tests  500 passed (500)

Before vs after (the key behavioral assertion)

  • Before: orchestration.run --worktree B while A is running → throws Coordinator already running: <A's run id>. runStop from B → stops A's coordinator.
  • After: getActiveCoordinatorRun('worktree:wt_b') does not see A's run (the orchestration-gates.test.ts test dispatches the real orchestration.run RPC for B and asserts an OK response with a distinct runId), and runStop {worktree:'worktree:wt_b'} leaves getActiveCoordinatorRun('worktree:wt_a') pointing at A's run.

Local gate

oxlint <13 changed files>                                   → clean (exit 0)
tsgo --noEmit -p config/tsconfig.node.json                   → clean (exit 0)
oxfmt --write <13 changed files>                             → formatted

Electron note

End-to-end two-worktree visual repro was not run: ports 9333–9336 were occupied by other worktrees' running dev builds, and attaching there risks driving the wrong instance. The regression is a main-process data-layer defect fully proven by the unit tests above; the validation plan marked Electron optional for this bug.

@nwparker

nwparker commented Jun 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Follow-up pushed in fa84418 (supersedes the c32174f tradeoff note):

  • Added explicit global coordinator mode: omitting --worktree or passing --worktree all starts the global coordinator, which still sees all tasks and therefore intentionally conflicts with scoped coordinators.
  • Tightened run-stop to exact scopes: --worktree <selector> stops only that workspace, --worktree all stops only the global coordinator, and bare run-stop succeeds only when exactly one coordinator is active.
  • Zero-tradeoff status: no remaining valid workflow loss found. A scoped coordinator owning only its workspace's tasks is the fixed contract; cross-worktree pickup was [Bug]: Multiple orchestrators in a single workspace kill each other #4389, not a supported workflow.

Validation:

  • pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/rpc/methods/orchestration-gates.test.ts (9 passed)
  • pnpm exec vitest run --config config/vitest.config.ts src/cli/index.test.ts -t "orchestration run|orchestration run-stop|Documented flag" (4 passed, 129 skipped)
  • pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/orchestration/db.test.ts -t "workspace scoping|coordinator runs" (8 passed, 49 skipped)
  • pnpm exec oxlint <8 changed files> (clean)
  • pnpm run typecheck:cli (clean)
  • pnpm run typecheck:node (clean)
  • git merge-tree --write-tree HEAD origin/main (clean merge tree)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/main/runtime/orca-runtime.ts (1)

17896-17905: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider routing this inline scoping through resolveCoordinatorRunForDispatch.

The workspace_key == null ? global : scoped decision is now duplicated here and in resolveCoordinatorRunForDispatch (Line 7711). Since the workspace-scoping fallback rule is the crux of this PR, keeping two copies risks future divergence (and silent re-introduction of the cross-attribution bug). The only reason this site can't reuse the helper today is that it must honor the locally-resolved db rather than this._orchestrationDb; threading an optional db into the helper would let both sites share one branch.

♻️ Optional: share one scoping branch
   private resolveCoordinatorRunForDispatch(
-    workspaceKey: string | null
+    workspaceKey: string | null,
+    db = this._orchestrationDb
   ): ReturnType<OrchestrationDb['getActiveCoordinatorRun']> {
-    if (!this._orchestrationDb) {
+    if (!db) {
       return undefined
     }
     if (workspaceKey == null) {
-      return this._orchestrationDb.getActiveCoordinatorRun()
+      return db.getActiveCoordinatorRun()
     }
-    return this._orchestrationDb.getActiveCoordinatorRunForWorkspace(workspaceKey)
+    return db.getActiveCoordinatorRunForWorkspace(workspaceKey)
   }

Then at Line 17900:

     const activeRun =
       dispatch.status === 'completed'
         ? undefined
-        : dispatch.workspace_key == null
-          ? db?.getActiveCoordinatorRun?.()
-          : db?.getActiveCoordinatorRunForWorkspace?.(dispatch.workspace_key)
+        : db
+          ? this.resolveCoordinatorRunForDispatch(dispatch.workspace_key, db)
+          : undefined

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 40cf347e-ea30-4ca6-bb84-d0aa3d324946

📥 Commits

Reviewing files that changed from the base of the PR and between e51b2df and fa84418.

📒 Files selected for processing (16)
  • skills/orchestration/SKILL.md
  • src/cli/handlers/orchestration.ts
  • src/cli/help.ts
  • src/cli/index.test.ts
  • src/cli/specs/orchestration.ts
  • src/main/runtime/orca-runtime.test.ts
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration/coordinator.ts
  • src/main/runtime/orchestration/db.test.ts
  • src/main/runtime/orchestration/db.ts
  • src/main/runtime/orchestration/formatter.test.ts
  • src/main/runtime/orchestration/types.ts
  • src/main/runtime/rpc/methods/orchestration-gates.test.ts
  • src/main/runtime/rpc/methods/orchestration-gates.ts
  • src/main/runtime/rpc/methods/orchestration.ts
  • src/main/runtime/runtime-rpc.test.ts

Comment thread src/cli/handlers/orchestration.ts Outdated
Comment on lines 141 to 147
if (resolvedWorktree !== undefined) {
const response = await client.call<{ handle: string }>('terminal.resolveActive', {
worktree: resolvedWorktree
})
return response.result.handle
}
return await getTerminalHandle(flags, cwd, client)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Don't fall back to getTerminalHandle() after normalizing --worktree all.

Line 482 intentionally clears terminalWorktree for the global coordinator, but Line 147 then re-reads the raw CLI flags through getTerminalHandle(). Because that helper resolves worktree state from flags, orca orchestration run --worktree all can still be treated like a concrete worktree selection instead of the unscoped/global case.

Suggested fix
-  if (resolvedWorktree !== undefined) {
+  if (resolvedWorktree !== undefined || getOptionalStringFlag(flags, 'worktree') === 'all') {
     const response = await client.call<{ handle: string }>('terminal.resolveActive', {
       worktree: resolvedWorktree
     })
     return response.result.handle
   }

Also applies to: 481-489

nwparker and others added 5 commits July 20, 2026 13:27
…ace (#4389)

Two orchestrators started in different worktrees of one Orca instance
shared a single unscoped orchestration.db, so they blocked each other's
start, stopped the wrong coordinator, counted/dispatched each other's
tasks, and cross-attributed worker terminals in the agent-status UI.

Add a nullable workspace_key column (schema v5 -> v6) to coordinator_runs,
tasks, dispatch_contexts, and messages, and thread a per-coordinator
workspace key through the writes and run-scoped reads. The start/stop
guard now scopes getActiveCoordinatorRun by workspace and keys live
coordinators in a Map instead of one module-level singleton, so a run in
one workspace can't orphan or stop another's. Runtime helpers attribute a
worker terminal to the coordinator that owns its dispatch's workspace.

Existing rows keep NULL workspace_key (legacy/global) and every scoped
read matches NULL, so single-orchestrator installs are unaffected. Scope
resolution goes through the existing worktree-selector resolution, so it
is provider-agnostic and works for SSH/remote runtimes.

Fixes #4389

Co-authored-by: Orca <help@stably.ai>
Rebase the workspace-scoped coordinator fix onto current main while preserving stale terminal handle validation. Treat every active coordinator prompt as user-owned so push-on-idle cannot auto-submit into an earlier workspace's coordinator when another run is newer, and index the handle lookup used by that path.
@OrcaWin
OrcaWin marked this pull request as draft October 27, 2026 09:52
@OrcaWin

OrcaWin commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Moving this PR to draft because it has been untouched for more than 80 days (latest recorded PR activity or commit timestamp: 2026-07-21T21:28:53Z; checked on 2026-10-27). This keeps the review queue focused on active work while preserving the proposal. Please mark it ready for review when work resumes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Multiple orchestrators in a single workspace kill each other

3 participants